Method and apparatus for adding a database partition
View Patent ↗A data repository system and method are provided. A method in accordance with an embodiment includes an operation that can be used to port data from one or more existing database partitions to new database partitions according to a minimally progressive hash. The method can be used to increase the overall size of databases while a system runs hot, with little or no downtime.
1. A method for adding a database partition comprising:
determining an existing number of partitions;
adding at least one new partition to said existing number of partitions;
examining each existing partition to determine which portion of each existing partition is to be transferred according to a minimally progressive hash operation; said minimally progressive hash operation configured to identify at least one portion to be transferred based on: a hash of an identifier of said at least one portion, said existing number of partitions and a second number of partitions resulting from adding said at least one new partition;
transitioning said at least one portion to said new partition according to the determination made during said examining step.
2. The method of claim 1 wherein said hashing operation is selected from a class of partitioning algorithms that substantially maintain a maximum, or substantially maximum, consistency between consecutive partition sizes.
3. The method of claim 1 wherein said method is repeated until a desired number of partitions have been added.
4. The method of claim 1 wherein said hashing operation is implemented according to the following pseudo-code:
define function part(hash, 1) -> (1, hash)
define function part(hash, size) -> (partition, remainder) as
let (partition, remainder) = part(hash, size−1)
if remainder mod size = 0 then
return (size, remainder / size)
else
return (partition, remainder * (size−1) / size)
end
end function
define function part(key) -> partition
return part(hashValueOf(key), current_partition_size) end function.
5. The method of claim 1 wherein said hashing operation is implemented according to the following pseudo-code:
define function part(hash, size) -> (partition, remainder) as
let (partition, remainder) = part(hash, size−1)
let new_LCM = lcm(1..size)
let old_LCM = lcm(1..size−1)
let number_per_old = new_LCM / size
let steals_per_old = number_per_old / (size−1)
let cycle = remainder / (new_LCM / (size−1))
let cycle_remainder = remainder mod (new_LCM / (size−1))
let cycle_steal = max(0, 1 + cycle_remainder − number_per_old)
if cycle_steal > 0 then
return (size, (partition − 1) * steals_per_old + cycle_steal −
1 + cycle * number_per_old)
else
return (partition, cycle * number_per_old + cycle_remainder)
end
end function.
6. A computer-based apparatus for adding a database partition comprising an interface for connection to an existing database; a processor configured to a determine an existing number of partitions in said database and to examine each existing partition in said database to further determine which portion of each existing partition of said database is to be transferred according to a minimally progressive hash operation; said processor further configured to add at least one new partition to said existing database and to transitioning portions of existing partitions to said new partition according to the determinations made during said examining step, wherein said minimally progressive hash operation is configured to identify at least one portion to be transferred based on: a hash of an identifier of said at least one portion, said existing number of partitions and a second number of partitions resulting from said adding at least one new partition to said existing number of partitions.
7. The apparatus of claim 6 wherein said hashing operation is selected from a class of partitioning algorithms that substantially maintain a maximum, or substantially maximum, consistency between consecutive partition sizes.
8. The apparatus of claim 6 wherein said hashing operation is implemented according to the following pseudo-code:
define function part(hash, 1) -> (1, hash)
define function part(hash, size) -> (partition, remainder) as
let (partition, remainder) = part(hash, size−1)
if remainder mod size = 0 then
return (size, remainder / size)
else
return (partition, remainder * (size−1) / size)
end
end function
define function part(key) -> partition
return part(hashValueOf(key), current_partition_size) end function.
9. The apparatus of claim 6 wherein said hashing operation is implemented according to the following pseudo-code:
define function part(hash, size) -> (partition, remainder) as
let (partition, remainder) = part(hash, size−1)
let new_LCM = lcm(1..size)
let old_LCM = lcm(1..size−1)
let number_per_old = new_LCM / size
let steals_per_old = number_per_old / (size−1)
let cycle = remainder / (new_LCM / (size−1))
let cycle_remainder = remainder mod (new_LCM / (size−1))
let cycle_steal = max(0, 1 + cycle_remainder −
number_per_old)
if cycle_steal > 0 then
return (size, (partition − 1) * steals_per_old +
cycle_steal − 1 + cycle * number_per_old)
else
return (partition, cycle * number_per_old + cycle_remainder)
end
end function.
10. A computer readable media storing a plurality of programming instructions; said programming instructions executable by a computing apparatus; said programming instructions configured to implement a method for adding a database partition comprising:
determining an existing number of partitions;
adding at least one new partition to said existing number of partitions;
examining each existing partition to determine which portion of each existing partition is to be transferred according to a minimally progressive hash operation;
said minimally progressive hash operation configured to identify at least one portion to be transferred based on: a hash of an identifier of said at least one portion, said existing number of partitions and a second number of partitions resulting from adding said at least one new partition;
transitioning portions of existing partitions to said new partition according to the determinations made during said examining step.
11. The computer readable media of claim 10 wherein said hashing operation is selected from a class of partitioning algorithms that substantially maintain a maximum, or substantially maximum, consistency between consecutive partition sizes.
12. The computer readable media of claim 10 wherein said method is repeated until a desired number of partitions have been added.
13. The computer readable media of claim 10 wherein said hashing operation is implemented according to the following pseudo-code:
define function part(hash, 1) −> (1, hash)
define function part(hash, size) −> (partition, remainder) as
let (partition, remainder) = part(hash, size-1)
if remainder mod size = 0 then
return (size, remainder/size)
else
return (partition, remainder * (size-1)/size)
end
end function
define function part( key) −> partition
return part(hashValueOf (key), current_partition_size) end
function.
14. The computer readable media of claim 10 wherein said hashing operation is implemented according to the following pseudo-code:
define function part(hash, size) -> (partition, remainder) as
let (partition, remainder) = part(hash, size−1)
let new_LCM = lcm(1..size)
let old_LCM = lcm(1..size−1)
let number_per_old = new_LCM / size
let steals_per_old = number_per_old / (size−1)
let cycle = remainder / (new_LCM / (size−1))
let cycle_remainder = remainder mod (new_LCM / (size−1))
let cycle_steal = max(0, 1 + cycle_remainder −
number_per_old)
if cycle_steal > 0 then
return (size, (partition − 1) * steals_per_old +
cycle_steal − 1 + cycle * number_per_old)
else
return (partition, cycle * number_per_old + cycle_remainder)
end
end function.