System and/or method for bulk loading of records into an ordered distributed database
In a large-scale transaction such as the bulk loading of new records into an ordered, distributed database, a transaction limit such as an insert limit may be chosen, partitions on overfull storage servers may be designated to be moved to underfull storage servers, and the move assignments may be based, at least in part on the degree to which a storage server is underfull and the move and insertion costs of the partitions to be moved.
1. A method comprising:
identifying a plurality of partitions on a first storage server, wherein at least some partitions of the plurality of partitions includes a set of existing records;
identifying a set of new records to be inserted into at least one partition of the plurality of partitions;
in response to determining the first storage server is in an overfull state, said determining being based at least in part on the identified set of new records exceeding an insert limit, said insert limit being based at least in part on a maximum permissible number of records the plurality of partitions is capable to receive, determining an insertion cost for inserting a portion of said set of new records into respective partitions of the plurality of partitions, wherein said insertion cost is based, at least in part, on a number of new records within the portion of said set of new records to be inserted into the respective partitions of the plurality of partitions;
for each of the respective partitions of the plurality of partitions, determining a move cost based, at least in part, on a number of records in each of the respective partitions of the set of existing records;
selecting a first partition of the respective partitions of the plurality of partitions for transition to a second storage server based, at least in part, on a ratio of said move cost associated with the first partition of the respective partitions of the plurality of partitions to said insertion cost associated with the first partition of the respective partitions of the plurality of partitions; and
transitioning the set of existing records of the selected first partition to the second storage server identified as an underfull storage server.
2. The method of claim 1 , and further comprising:
determining whether said first storage server is in an overfull state based at least in part on an insert limit, wherein said insert limit is based at least in part on a maximum permissible number of records that said plurality of partitions is capable to receive.
3. The method of claim 1 further comprising determining a cost based at least in part on a sum of a maximum permissible number of new records that said second storage server is to receive, and a maximum permissible number of existing records that said first storage server is to transfer.
4. The method of claim 1 wherein said ratio is a first ratio, the method further comprising:
selecting a second partition of the respective partitions of the plurality of partitions for transition to the second storage server based, at least in part, on a second ratio, wherein said second ratio comprises a ratio of a move cost associated with the second partition of the respective partitions of the plurality of partitions to said insertion cost; and
determining an order for transitioning the first partition of the respective partitions of the plurality of partitions relative to the second partition of the respective partitions of the plurality of partitions based at least in part on said first ratio and said second ratio.
5. An apparatus comprising:
a computing platform, the computing platform comprising at least one communications port for communicating with storage servers, the computing platform to:
identify a plurality of partitions on a first storage server, wherein at least some partitions of the plurality of partitions includes a set of existing records;
identify a set of new records to be inserted into at least one partition of the plurality of partitions;
in response to determining the first storage server is in an overfull state, said determining being based at least in part on the identified set of new records exceeding an insert limit, said insert limit being based at least in part on a maximum permissible number of records the plurality of partitions is capable to receive, determine an insertion cost for inserting a portion of said set of new records into respective partitions of the plurality of partitions, wherein said insertion cost is based, at least in part, on a number of new records within the portion of said set of new records to be inserted into the respective partitions of the plurality of partitions;
for each of the respective partitions of the plurality of partitions, determine a move cost based, at least in part, on a number of records in each of the respective partitions of the set of existing records;
select a first partition of the respective partitions of the plurality of partitions for transition to a second storage server based, at least in part, on a ratio of said move cost associated with the first partition of the respective partitions of the plurality of partitions to said insertion cost associated with the first partition of the respective partitions of the plurality of partitions; and
transition the set of existing records of the selected first partition to the second storage server identified as an underfull storage server.
6. The apparatus of claim 5 , the computing platform to further:
determine whether said first storage server is in an overfull state based at least in part on an insert limit, wherein said insert limit is based at least in part on a maximum permissible number of records that said plurality of partitions is capable to receive.
7. The apparatus of claim 5 wherein said ratio is a first ratio, the computing platform to further:
select a second partition of the respective partitions of the plurality of partitions for transition to the second storage server based, at least in part, on a second ratio, wherein said second ratio comprises a ratio of a move cost associated with the second partition of the respective partitions of the plurality of partitions to said insertion cost; and
determine an order for transitioning the first partition of the respective partitions of the plurality of partitions relative to the second partition of the respective partitions of the plurality of partitions based at least in part on said first ratio and said second ratio.
8. An article comprising:
a non-transitory storage medium comprising machine readable instructions stored thereon executable by a computing platform to:
identify a plurality of partitions on a first storage server, wherein at least some partitions of the plurality of partitions includes a set of existing records;
identify a set of new records to be inserted into at least one partition of the plurality of partitions;
in response to determining the first storage server is in an overfull state, said determining being based at least in part on the identified set of new records exceeding an insert limit, said insert limit being based at least in part on a maximum permissible number of records the plurality of partitions is capable to receive, determine an insertion cost for inserting a portion of said set of new records into respective partitions of the plurality of partitions, wherein said insertion cost is based, at least in part, on a number of new records within the portion of said set of new records to be inserted into the respective partitions of the plurality of partitions;
for each of the respective partitions of the plurality of partitions, determine a move cost based, at least in part, on a number of records in each of the respective partitions of the set of existing records;
select a first partition of the respective partitions of the plurality of partitions for transition to a second storage server based, at least in part, on a ratio of said move cost associated with the first partition of the respective partitions of the plurality of partitions to said insertion cost associated with the first partition of the respective partitions of the plurality of partitions; and
transition the set of existing records of the selected first partition to the second storage server identified as an underfull storage server.
9. The article of claim 8 , said machine readable instructions being further executable by said computing platform to:
determine whether said first storage server is in an overfull state based at least in part on an insert limit, wherein said insert limit is based at least in part on a maximum permissible number of records that said plurality of partitions is capable to receive.
10. The article of claim 8 , said machine readable instructions being further executable by said computing platform to:
determine a cost based at least in part on a sum of a maximum permissible number of new records that said second storage server is to receive, and a maximum permissible number of existing records that said first storage server is to transfer.
11. The article of claim 8 wherein said ratio is a first ratio, said machine readable instructions being further executable by said computing platform to:
select a second partition of the respective partitions of the plurality of partitions for transition to the second storage server based, at least in part, on a second ratio, wherein said second ratio comprises a ratio of a move cost associated with the second partition of the respective partitions of the plurality of partitions to said insertion cost; and
determine an order for transitioning the first partition of the respective partitions of the plurality of partitions relative to the second partition of the respective partitions of the plurality of partitions based at least in part on said first ratio and said second ratio.