Partitioning data into chunk groupings for use in a dispersed storage network
A computing device for use in a storage network includes one or more network interfaces, memory with operational instructions and a processing module. The processing module is configured to execute the operational instructions receive a storage request for data and select a plurality of memory devices for storage of the data. The processing module partitions the data into a plurality of chunk sets and encode each chunk set of the plurality of chunk sets according to a dispersed storage error coded function to produce a plurality of chunks. Finally, the processing module facilitates storage of each chunk of the plurality of chunks in a memory device of the plurality of memory devices.
1 . A computing device of a network storage, the computing device comprises:
one or more network interfaces;
memory including operational instructions; and
a processing module operably coupled to the memory and the one or more network interfaces, the processing module configured to execute the operational instructions to:
receive a storage request for data;
determine a data type for the data;
based on the data type, determine whether to partition the data into variable-sized data blocks:
in response to a determination to divide the data into variable-sized data blocks, partition the data according to a partition function into a plurality of data blocks;
encode each data block of the plurality of data blocks according to a dispersed storage error coded function to produce a plurality of data fragments;
facilitate storage of the plurality of data fragments in a plurality of storage network node.
2 . The computing device of claim 1 , wherein the processing module is further configured to execute the operational instructions to select the plurality of memory devices for storage of the data based on a capability level associated with each of the memory devices.
3 . The computing device of claim 1 , wherein the processing module is further configured to execute the operational instructions to select the plurality of memory devices for storage of the data based on at least one of:
an estimated distributed computing loading level;
a memory device capability indicator;
a memory device performance indicator;
a memory device availability level indicator;
a task schedule, or a memory device threshold computing capability indicator.
4 . The computing device of claim 1 , wherein the processing module is further configured to execute the operational instructions to:
select specific memory devices of the plurality of memory devices for storage of the data when memory availability level indicators for the specific memory devices compare favorably to an estimated distributed computing loading level.
5 . The computing device of claim 1 , wherein the processing module is further configured to execute the operational instructions to:
generate each data block of the plurality of data blocks in accordance with a data block size and processing parameters.
6 . The computing device of claim 5 , wherein the processing module is further configured to execute the operational instructions to:
generate a data block size for each data block, wherein the data block size is generated based on at least one of: a data block size selection scheme, a predetermination, or receiving the data block size.
7 . The computing device of claim 5 , wherein the data block size is greater than or equal to a largest data group size such that a largest data group associated with the largest data group size fits within any data block set.
8 . The computing device of claim 1 , wherein the storage network node includes a plurality of memory devices, wherein the plurality of memory devices includes one or more solid state memories.
9 . The computing device of claim 1 , wherein an unused capacity within each data block is padded with pad bytes, and wherein each pad byte includes at least one of: a predetermined value, a random value, a value associated with a data block number, a value associated with at least one data group packed into the data block, or at least one partial task associated with the data block.
10 . The computing device of claim 1 , wherein the processing module is further configured to execute the operational instructions to:
in response to a determination not to divide the data into variable-sized data blocks, partition the data according to another partition function into another plurality of data blocks;
encode each data block of the another plurality of data blocks according to a dispersed storage error coded function to produce a plurality of data fragments;
facilitate storage of the plurality of data fragments in a plurality of storage network node.
11 . The computing device of claim 1 , wherein the plurality of data fragments is a decode threshold number of fragments.
12 . A method for storing data in a data storage network comprises
receiving a storage request for data
determining a data type for the data;
based on the data type, determining whether to partition the data into variable-sized data blocks;
in response to a determination to divide the data into variable-sized data blocks, partitioning the data according to a partition function into a plurality of data blocks;
encoding each data block of the plurality of data blocks according to a dispersed storage error coded function to produce a plurality of data fragments; and
facilitating storage of the plurality of data fragments in a plurality of storage network node.
13 . The method of claim 12 , further comprising:
selecting the plurality of memory devices for storage of the data based on a capability level associated with each of the memory devices.
14 . The method of claim 12 , further comprising:
selecting the plurality of memory devices for storage of the data based on at least one of:
an estimated distributed computing loading level;
a memory device capability indicator;
a memory device performance indicator;
a memory device availability level indicator;
a task schedule; and
a memory device threshold computing capability indicator.
15 . The method of claim 14 , wherein the plurality of memory devices includes at a solid state memory.
16 . The method of claim 12 , further comprising:
selecting specific memory devices for storage of the data when memory availability level indicators for the specific memory devices compare favorably to an estimated distributed computing loading level.
17 . The method of claim 12 , further comprising:
generating a data block size for each data block, wherein the data block size is generated based on at least one of: a data block size selection scheme, a predetermination, or receiving the data block size.
18 . A computing device comprises:
one or more network interfaces;
memory including operational instructions; and
a processing module operably coupled to the memory and the one or more network interfaces, the processing module configured to execute the operational instructions to:
receive a storage request for data;
determine a data type for the data;
based on the data type, determining whether to partition the data into variable-sized data blocks;
in response to a determination to divide the data into variable-sized data blocks, partition the data according to a partition function into a plurality of data blocks;
encode each data block of the plurality of data blocks according to a dispersed storage error coded function to produce a plurality of data fragments;
select a plurality of solid-state memory devices for storage of the plurality of data fragments;
facilitate storage of each data fragment of the plurality of data blocks the plurality of data fragments in a memory device of the plurality of solid-state memory devices.