Elastic sharding of data in a multi-tenant cloud

1. A method comprising:

receiving, via a computer network, first primary data from a first tenant;

receiving first auxiliary data relating to the first primary data from the first tenant;

creating, using one or more hardware processors, a first index for the first primary data and the first auxiliary data from the first tenant;

storing, in a memory, the first index as a first shardlet;

bundling the first shardlet with one or more other shardlets for the first tenant in a first shard group for the first tenant, causing the first shard group to contain a plurality of shardlets for the first tenant;

receiving second primary data from a second tenant in the computer network;

receiving second auxiliary data relating to the second primary data from the second tenant;

creating, using the one or more hardware processors, a second index for the second primary data and the second auxiliary data from the second tenant;

storing, in the memory, the second index as a second shardlet;

bundling the second shardlet with one or more other shardlets for the second tenant in a second shard group, causing the second shard group to contain a plurality of shardlets for the second tenant;

packing the first shard group and second shard group in a first shard, causing the first shard to contain the plurality of shardlets for the first tenant and the plurality of shardlets for the second tenant;

storing the first shard in a first instance of a distributed database, the distributed database comprising a plurality of instances, each instance operating on a different logical or physical device;

distributing the first shard to a first search core;

distributing a second shard to a second search core;

receiving a first search request;

determining that the first search request involves data located at an index included as a shardlet in a shard group in the first shard; and

in response to the determining, forwarding the first search request to the first search core in lieu of the second search core.

2. The method of claim 1 , wherein the packing is performed dynamically in response to one or more updates to the first primary data, the second primary data, the first auxiliary data, or the second auxiliary data.

3. The method of claim 1 , wherein the packing the first shard group and second shard group is performed dynamically based on weighting scores assigned to each of one or more shards, the weighting scores dynamically assigned to each of the one or more shards based on one or more factors of the first tenant.

4. The method of claim 3 , wherein the one or more factors include size of data of the first tenant.

5. The method of claim 3 , wherein the one or more factors include a weight applied to a size of the first tenant.

6. The method of claim 3 , wherein the one or more factors include number of replicas for the first tenant.

7. The method of claim 3 , wherein the weighting scores are further based on one or more factors of other tenants.

8. A system comprising:

one or more hardware processors; and

a computer-readable medium storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations comprising:

receiving, via a computer network, first primary data from a first tenant;

receiving first auxiliary data relating to the first primary data from the first tenant;

creating, using one or more hardware processors, a first index for the first primary data and the first auxiliary data from the first tenant;

storing, in a memory, the first index as a first shardlet;

bundling the first shardlet with one or more other shardlets for the first tenant in a first shard group for the first tenant, causing the first shard group to contain a plurality of shardlets for the first tenant;

receiving second primary data from a second tenant in the computer network;

receiving second auxiliary data relating to the second primary data from the second tenant;

creating, using the one or more hardware processors, a second index for the second primary data and the second auxiliary data from the second tenant;

storing, in the memory, the second index as a second shardlet;

bundling the second shardlet with one or more other shardlets for the second tenant in a second shard group, causing the second shard group to contain a plurality of shardlets for the second tenant;

packing the first shard group and second shard group in a first shard, causing the first shard to contain the plurality of shardlets for the first tenant and the plurality of shardlets for the second tenant;

storing the first shard in a first instance of a distributed database, the distributed database comprising a plurality of instances, each instance operating on a different logical or physical device;

distributing the first shard to a first search core;

distributing a second shard to a second search core;

receiving a first search request;

determining that the first search request involves data located at an index included as a shardlet in a shard group in the first shard; and

in response to the determining, forwarding the first search request to the first search core in lieu of the second search core.

9. The system of claim 8 , wherein the packing is performed dynamically in response to one or more updates to the first primary data or the first auxiliary data.

10. The system of claim 8 , wherein the packing is performed dynamically based on weighting scores assigned to each of one or more shards, the weighting scores dynamically assigned to each of the one or more shards based on one or more factors of the first tenant.

11. The system of claim 10 , wherein the one or more factors include size of data of the first tenant.

12. The system of claim 10 , wherein the one or more factors include a weight applied to a size of the first tenant.

13. The system of claim 10 , wherein the one or more factors include number of replicas for the first tenant.

14. The system of claim 10 , wherein the weighting scores are further based on one or more factors of other tenants.

15. A non-transitory machine-readable storage medium comprising instructions, which when implemented by one or more machines, cause the one or more machines to perform operations comprising:

receiving, via a computer network, first primary data from a first tenant;

receiving first auxiliary data relating to the first primary data from the first tenant;

creating, using one or more hardware processors, a first index for the first primary data and the first auxiliary data from the first tenant;

storing, in a memory, the first index as a first shardlet;

bundling the first shardlet with one or more other shardlets for the first tenant in a first shard group for the first tenant, causing the first shard group to contain a plurality of shardlets for the first tenant;

receiving second primary data from a second tenant in the computer network;

receiving second auxiliary data relating to the second primary data from the second tenant;

creating, using the one or more hardware processors, a second index for the second primary data and the second auxiliary data from the second tenant;

storing, in the memory, the second index as a second shardlet;

bundling the second shardlet with one or more other shardlets for the second tenant in a second shard group, causing the second shard group to contain a plurality of shardlets for the second tenant;

packing the first shard group and second shard group in a first shard, causing the first shard to contain the plurality of shardlets for the first tenant and the plurality of shardlets for the second tenant;

storing the first shard in a first instance of a distributed database, the distributed database comprising a plurality of instances, each instance operating on a different logical or physical device;

distributing the first shard to a first search core;

distributing a second shard to a second search core;

receiving a first search request;

determining that the first search request involves data located at an index included as a shardlet in a shard group in the first shard; and

in response to the determining, forwarding the first search request to the first search core in lieu of the second search core.

16. The non-transitory machine-readable storage medium of claim 15 , wherein the packing is performed dynamically in response to one or more updates to the first primary data, the second primary data, the first auxiliary data, or the second auxiliary data.

17. The non-transitory machine-readable storage medium of claim 15 , wherein the packing is performed dynamically based on weighting scores assigned to each of one or more shards, the weighting scores dynamically assigned to each of the one or more shards based on one or more factors of the first tenant.

18. The non-transitory machine-readable storage medium of claim 17 , wherein the one or more factors include size of data of the first tenant.

19. The non-transitory machine-readable storage medium of claim 17 , wherein the one or more factors include a weight applied to a size of the first tenant.

20. The non-transitory machine-readable storage medium of claim 17 , wherein each shardlet is a Lucene index.